Word Alignment and the Expectation-Maximization Algorithm

The purpose of this tutorial is to give you an example of how to take a simple discrete probabilistic model and derive the expectation maximization updates for it and then turn them into code. We give some examples and identify the key ideas that make the algorithms work. These are meant to be as intuitive as possible, but for those curious about the underlying mathematics, we also provide some derivations, and point the reader to additional tutorials when mathematical depth goes beyond the scope of the tutorial. But you needn’t follow these derivations in detail to understand the main concepts, so whether you try to is to is up to you. We’ll consider IBM Model 1, which has been the subject of other similar tutorials [Collins(2011), Knight(1999)]. A great deal of the material here is inspired, either directly or indirectly, by these papers, as well as the concise original description in Sections 4 and 4.1 of [Brown et al.(1993)], and of course, Chapter 4 of Philipp Koehn’s textbook (2010) . Everyone learns differently, though, so I urge you to take a look at one of those if you’re still confused after reading this one. The basic plan is to learn a conditional probabilistic model of a French sentence f given an English sentence e, which we’ll denote pθ(f|e). Subscript θ refers to the set of parameters in the model; we’ll describe what it looks like shortly. We also have a dataset D of N sentence pairs that are known to be translations of each other, D = {(f(1), e(1))...(f(N), e(N))}, where each superscript (n) indexes a different sentence pair. The object of our model will be to uncover the hidden word-to-word correspondences in these translation pairs. We will learn the model from this data, and once we’ve learned it, we’ll use it to predict the existence of the missing word alignments.